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DETAILED ACTION 

Re-open Prosecution after Appeal Brief 

1 . In view of the supplemental appeal brief filed on 08/1 9/09, PROSECUTION IS 
HEREBY REOPENED. A new ground of rejection is set forth below. 

To avoid abandonment of the application, appellant must exercise one of the 
following two options: 

(1 ) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply 
under 37 CFR 1.113 (if this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41 .31 followed 
by an appeal brief under 37 CFR 41 .37. The previously paid notice of appeal fee and 
appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth 
in 37 CFR 41 .20 have been increased since they were previously paid, then appellant 
must pay the difference between the increased fees and the amount previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by 
signing below. 

/Richemond Dorvil/ 

Supervisory Patent Examiner, Art Unit 2626 

Response to Arguments 

2. Applicant's arguments with respect to claims 1 - 5, 7 - 1 1 , 13 - 27, 29 -33, and 
35 - 46 have been considered but are moot in view of the new ground(s) of rejection. 
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Appellants argue that Coffman does not teach means for establishing a user 
preference value for each input and output port from properties, said properties 
comprising the utilization made by a user of each input and output port (Appeal Brief, 
pages 13-19). 

Claim Rejections - 35 USC § 101 

3. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 23 - 27, 29 -33, and 35 - 46 are rejected under 35 USC 101 as 
not falling within one of the four statutory categories of invention. While the 
claims recite a series of steps to be performed, a statutory process under 35 
USC 101 must be tied to another statutory category (such as a manufacture or a 
machine) or transform underlying subject matter (such as an article or material) 
to a different state or thing. The steps in those claims can be performed 
manually without the use of a particular machine. Those claims could be 
interpreted as an interaction between two users, wherein inputs/outputs 
information are entered using different modalities. Thus, claims 13-27, 29 -33, 
and 35 - 44 do not define a statutory process. 
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Claim Rejections - 35 USC § 103 

4. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

5. Claims 1 -5, 7 - 1 1 , 13 - 27, 29 - 33, 35-46 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Coffman et al., (US PAP 2003/0005174) in view of 
Gergic et al., (US PAP 2003/0046316). 

As per claims 1 , and 23, Coffman et al., teach an interactive dialogue that 
comprises: 

at least one input port; two or more output ports ("converts the abstract output 
event into one or more modalities for presentation to the user"; paragraph 22, lines 6 - 
8; paragraph 60, lines 9-12); 

means for processing input responses to determine the semantic meaning 
thereof (paragraph 94, line 5); 

and control means for determining a suitable output prompt to be output from at 
least one of said output ports in response to a received input response ("sending output 
events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153); 

wherein said output ports are respectively arranged to output prompts of different 
types ("pen recognition, speech recognition, TTS"; paragraph 153); 

a first store storing input and output type data indicative of one or more 
properties of the input and output ports and/or the input responses and output prompts 
communicated there through ("The DMA will also manage the output of these 
methods by passing it to the appropriate components for processing, using an 
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algorithm string, similar to the algorithm string used for input handling (as 
described below), to control the response processing and generation by the 
appropriate engines... the algorithm string may comprise: front end+speech 
recognition+NLU. If the user input is a typed command, the algorithm string may 
be just NLU, etc"; paragraphs 53, and 60); 

wherein said input and output type data is updated when: i) any of said one or 
more properties change; and/or ii) output prompts are sent; and/or iii) input responses 
are received (The DMA will also manage the output of these methods by passing 
it to the appropriate components for processing, using an algorithm string, 
similar to the algorithm string used for input handling (as described below), to 
control the response processing and generation by the appropriate engines. ..if 
the user input comprises spoken utterances (voice command), the algorithm string 
may comprise: front end+speech recognition+NLU. If the user input is a typed 
command, the algorithm string may be just NLU, etc"; paragraphs 53, and 60). 

However, Coffman et al., do not specifically teach wherein one of said properties 
is the utilization made by a user of each input and output port; and means for 
establishing from said properties for each of said input and output ports a user 
preference value. 

Gergic et al., disclose that NL is a statement which is not limited to speech but 
encompasses all aspects of a natural multi-modal conversational application. It 
combines NL inputs with natural multi-modal input... any input is modeled 
independently of the modality as an input/output event that is then processed by 
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a dialog manager and arbitrator that will use history, dialog context and other 
meta-information (e.g., user preference, information about the device and 
application) to determine the target of the input event and/or engage a dialog with 
the user to complete, confirm, correct or disambiguate he intention of the user 
prior to executing the requested action (paragraph 23). 

Therefore, it would have been obvious to ordinary skill in the art at the time the 
invention was made to determine inputs/outputs' user preference as taught by Gergic et 
al., in Coffman et al., so that multiple user interfaces, e.g., GUI, speech, etc., are 
synchronized and continuously updated as a user interactively proceeds with one or the 
other modality (paragraph 21 , last four lines). 

As per claims 2, and 24, Coffman et al., teach an interactive dialogue that 
comprises: 

two or more input ports; at least one output port ("voice command, and typed 
command"; paragraph 60; paragraph 22); 

means for processing input responses received at one or more of said input ports 
to determine the semantic meaning thereof(paragraph 94, line 5); 

and control means for determining a suitable output prompt to be output from at 
least one of said output ports in response to a received input response ("sending output 
events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153); 
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wherein said input ports are respectively arranged to receive input responses of 
different types ("voice command, and typed command"; paragraph 60); the apparatus 
and method further comprising 

a first store storing input and output type data indicative of one or more 
properties of the input and output ports and/or the input responses and output prompts 
communicated there through ("The DMA will also manage the output of these 
methods by passing it to the appropriate components for processing, using an 
algorithm string, similar to the algorithm string used for input handling (as 
described below), to control the response processing and generation by the 
appropriate engines... the algorithm string may comprise: front end+speech 
recognition+NLU. If the user input is a typed command, the algorithm string may 
be just NLU, etc"; paragraphs 53, and 60); 

wherein said input and output type data is updated when: i) any of said one or 
more properties change; and/or ii) output prompts are sent; and/or iii) input responses 
are received ("The DMA will also manage the output of these methods by passing 
it to the appropriate components for processing, using an algorithm string, 
similar to the algorithm string used for input handling (as described below), to 
control the response processing and generation by the appropriate engines. ..if 
the user input comprises spoken utterances (voice command), the algorithm string 
may comprise: front end+speech recognition+NLU. If the user input is a typed 
command, the algorithm string may be just NLU, etc"; paragraphs 53, and 60). 
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However, Coffman et al., do not specifically teach wherein one of said properties 
is the utilization made by a user of each input and output port; and means for 
establishing from said properties for each of said input and output ports a user 
preference value. 

Gergic et al., disclose that NL is a statement which is not limited to speech but 
encompasses all aspects of a natural multi-modal conversational application. It 
combines NL inputs with natural multi-modal input... any input is modeled 
independently of the modality as an input/output event that is then processed by 
a dialog manager and arbitrator that will use history, dialog context and other 
meta-information (e.g., user preference, information about the device and 
application) to determine the target of the input event and/or engage a dialog with 
the user to complete, confirm, correct or disambiguate he intention of the user 
prior to executing the requested action (paragraph 23). 

Therefore, it would have been obvious to ordinary skill in the art at the time the 
invention was made to determine inputs/outputs' user preference as taught by Gergic et 
al., in Coffman et al., so that multiple user interfaces, e.g., GUI, speech, etc., are 
synchronized and continuously updated as a user interactively proceeds with one or the 
other modality (paragraph 21 , last four lines). 

As per claims 3 and 25, Coffman et al., in view of Gergic et al., further disclose at 
least one additional output port, wherein said control means is further arranged to 
determine a suitable output prompt to be output from at least one of said output ports in 
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response to a received input response ("sending output events to the appropriate 
engine"); and wherein said output ports are respectively arranged to output prompts of 
different types (Coffman et al., "pen recognition, speech recognition, TTS"; paragraph 
102, lines 4-7; paragraph 153). 

As per claims 4 and 26, Coffman et al., in view of Gergic et al., further disclose 
that for any particular received input prompt, output prompts which are semantically 
synonymous ("semantic meaning") or which mutually contribute towards a single 
semantic message independent of type are output from two or more of the output ports 
(Coffman et al., "converts the abstract output event into one or more modalities for 
presentation to the user"; paragraph 94, line 5; paragraph 22, lines 6 - 8). 

As per claims 5 and 27, Coffman et al., in view of Gergic et al., further disclose 
that each input or output port is adapted to connect to one or more input or output 
devices via respective device gateways (Coffman et al., "input and output devices"; 
paragraph 172; paragraph 101, line 2). 

As per claims 7, and 29, Coffman et al., in view of Gergic et al., further disclose 
that one of said properties is the connection of appropriate input or output devices to 
each of said input or output ports (Coffman et al., "properties includes the resources the 
application needs for processing the user input"; paragraph 60). 
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As per claims 8, and 30, Coffman et al., in view of Gergic et al., further disclose 
that one of said properties is user preference value for each of said input and output 
ports (Gergic et al., "a dialog manager and arbitrator that will use history, dialog 
context and other meta-information (e.g., user preference, information about the 
device and application) to determine the target of the input event and/or engage a 
dialog with the user to complete, confirm, correct or disambiguate he intention of 
the user prior to executing the requested action"; paragraph 23). 

As per claims 9, and 31 , Coffman et al., in view of Gergic et al., further disclose 
that one of said properties is device property data of input or output devices connected 
to said input or output ports (Coffman et al., "voice command and typed command"; 
paragraph 60, lines 9-12). 

As per claims 10 and 32, Coffman et al., in view of Gergic et al., further disclose 
that one of said properties is implementation data indicative of: whether an output 
prompt has been implemented in each output prompt type and/or input parse rules for 
each input response type (Coffman et al., "NLU parse tree... and data associated with a 
NLU process"; paragraph 103, lines 1 -4; paragraph 122). 

As per claims 1 1 and 33, Coffman et al., in view of Gergic et al., further disclose 
that one of said properties is type-supported data indicative of whether the apparatus is 
capable of receiving and/or outputting input responses and/or output prompts of each 
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type (Coffman et al., "converts the abstract output event into one or more modalities for 
presentation to the user"; paragraph 22). 

As per claims 13, and 35, Coffman et al., in view of Gergic et al., further disclose 
that the update of said data comprises instantiating new data structures ("hierarchical 
tree structure") to store the values of said changed properties, and storing said previous 
data to give a historical record of said data (Coffman et al., "transaction history"; 
paragraph 13, line 7; paragraph 64, lines 1 -4). 

As per claims 14, and 36, Coffman et al., in view of Gergic et al., further disclose 
that said input and output type data further includes timing data indicative of the timings 
of changes in said one or more properties (Coffman et al., "time stamped"; paragraph 
163). 

As per claims 15, and 37, Coffman et al., in view of Gergic et al., further disclose 
that said input and output type data comprises a single data entry for each input and 
output type, the value taken by a particular data entry being dependent on previous 
values of any one or more of that or other data entries (Coffman et al., "top scoring 
query result"; paragraphs 66, and 122). 

As per claims 16, and 38, Coffman et al., in view of Gergic et al., further disclose 
a second store data defining a dialogue to be held with a user, and dialogue 
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progression conditions which must be met to allow a user to progress through the 
dialogue, at least some of said conditions involving the stored input and output type 
data (Coffman et al., "for any given user input, arbitration mechanism will determine the 
target DMA instance managing the associated sub-dialog"; paragraphs 48, 49, and 59). 

As per claims 17, and 39, Coffman et al., in view of Gergic et al., further disclose 
a second store storing data defining a dialogue model comprising an initial state, a 
plurality of subsequent states, possible transitions between said states, and for each 
transition at least one associated condition to be satisfied before that transition is 
deemed allowable, at least some of said conditions involving the stored input and output 
type data (Coffman et al., "hierarchical tree structure that contains root, parent and 
children nodes"; paragraph 13; paragraph 49). 

As per claims 18, and 40, Coffman et al., in view of Gergic et al., further disclose 
that the second store comprises a plurality of distributed storage media (Coffman et al., 
"multimedia streams"; paragraph 104, line 15) 

As per claims 19, and 41 , Coffman et al., in view of Gergic et al., further disclose 
port control means for controlling the connections of input or output devices to said input 
or output ports in response to the stored input and output type data (Coffman et al., "a 
mechanism for sending output events to the appropriate engine"; paragraph 153, lines 4 
-6). 
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As per claims 20, and 42, Coffman et al., in view of Gergic et al., further disclose 
means for generating output prompts ("compose prompts"), said means being operable 
to generate output prompts adapted for particular output ports in dependence on the 
stored input and output type data (Coffman et al., "a mechanism for sending output 
events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153, lines 4 - 
6). 

As per claims 21 , and 43, Coffman et al., in view of Gergic et al., further disclose 
that first store comprises a plurality of distributed storage media each logically 
interconnected (Coffman et al., "multimedia streams"; paragraph 104, line 15). 

As per claims 22, and 44, Coffman et al., in view of Gergic et al., further disclose 
that the different types of output prompts or input responses comprise audio prompts or 
responses, or visual prompts or responses, or motor prompts or responses, in any 
combination thereof (Coffman et al., "text-to-display or prompt is provided"; paragraph 
166, lines 17, and 18). 

As per claim 45, and 46, Coffman et al., in view of Gergic et al., further disclose 
that a computer program or suite of programs so arranged such that when loaded into a 
computer it or they renders the computer an apparatus according to claims 1 , and 23 
(Coffman et al., paragraph 180). 
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Conclusion 

6. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Please see PTO-892 Form. 

7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to LEONARD SAINT CYR whose telephone number is 
(571) 272-4247. The examiner can normally be reached on Mon- Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on (571) 272-7602. The fax phone 
number for the organization where this application or proceeding is assigned is (571)- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or (571) 272-1000. 
LS 

11/19/09 

/Richemond Dorvil/ 
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